A computational-graph partitioning method for training memory-constrained DNNs

Many state-of-the-art Deep Neural Networks (DNNs) have substantial memory requirements. Limited device becomes a bottleneck when training those models. We propose ParDNN, an automatic, generic, and non-intrusive partitioning strategy for DNNs that are represented as computational graphs. ParDNN decides placement of DNN’s underlying graph operations across multiple devices so the devices’ constraints met time is minimized. completely independent deep learning aspects DNN. It requires no modification neither at model nor systems level implementation its operation kernels. partitions having billions parameters hundreds thousands in seconds to few minutes. Our experiments with TensorFlow on 16 GPUs demonstrate efficient 5 very large models while achieving superlinear scaling both batch size throughput. either outperforms or qualitatively improves upon related work.